Evaluating Students ' Assignments by Assessing How Well Students Test Their Own Code

Problem and Motivation: Testing accounts for 50% cost of software development. Because of the necessity of testing, educators are including more and more software testing[1] in programming [2] and software engineering courses [3, 4]. Current classroom assessment systems (e.g.,Web-CAT, ASSYST, Marmoset) use code coverage to evaluate how well students test their own code. Code coverage measures the percentage of code—e.g., branches or statements—that are executed during a test. It may falsely indicate test quality because code execution may not be initiated from a test oracle (e.g., assert statement) and the solution code may be partial, incomplete, or incorrect. Two robust and thorough mechanisms for evaluating student written tests are: 1) all-pairs student testing, and 2) mutation testing. Even though they are strong indicators of test quality and adequacy, because of technical difficulties they are rarely used for assessing student-written tests, especially when programs are written in object-oriented languages such as Java.